The training accuracy of a classification
model is much less important than how
well the model will work when given new unseen data. After all, we train models
so that they can be used on new data we find
in the real-world. After we have trained a
classification model, we should evaluate how it performs on a set
of new unseen data. In the last lesson, we created a model that would
predict whether a patient had diabetes or not based on
their blood glucose level. Now when applied to
some data that wasn't part of the training
set, we get predictions. Recall that X refers to
blood glucose levels, Y refers to whether
they're actually diabetic, and Y hat refers to the model's prediction as to whether they're diabetic or not. Simply calculating
how many predictions were correct is sometimes misleading or too
simplistic for us to understand the kind of errors it will make
in the real world. To get more detailed
information, we can tabulate the results in a structure called
a confusion matrix. The confusion matrix shows the total number of cases where, the model predicted zero and the actual label is
zero, true negatives, the model predicted one and the actual label is
one, true positives, the model predicted zero and
the actual label is one, false negatives, the
model predicted one and the actual label is
zero, false positives. The cells in the confusion
matrix are often shaded so that higher
values have a deeper shade. This makes it easier to see a strong diagonal trend from
top left to bottom right, highlighting the cells where the predicted value and
actual value are the same. From these core values, you can calculate a
range of other metrics that can help you evaluate
the performance of the model. For example, accuracy, of all the predictions, how many were correct? Recall, of all the cases
that are positive, how many did the model identify? Precision, of all the cases that the model predicted
to be positive, how many actually are positive?